Skip to content

test(index): make flaky IVF_RQ recall test deterministic#7679

Open
wombatu-kun wants to merge 1 commit into
lance-format:mainfrom
wombatu-kun:fix/deterministic-ivf-rq-recall-test
Open

test(index): make flaky IVF_RQ recall test deterministic#7679
wombatu-kun wants to merge 1 commit into
lance-format:mainfrom
wombatu-kun:fix/deterministic-ivf-rq-recall-test

Conversation

@wombatu-kun

@wombatu-kun wombatu-kun commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

What

test_build_ivf_rq (rust/lance/src/index/vector/ivf/v2.rs) is flaky: it builds an IVF_RQ index over seeded random vectors and asserts recall >= 0.5, but recall sits right on the bar and occasionally dips below. It surfaced in CI on #7371 as recall: 0.49 for the case_2::rotation_type_1_RQRotationType__Fast permutation (nlist=1, Cosine, Fast rotation) - see the failed linux-build job: https://github.com/lance-format/lance/actions/runs/28875488220/job/85649319644. It is a pre-existing flake, not caused by any recent change to the search path.

Root cause

The test data is already deterministic (generate_random_array_with_range seeds StdRng::from_seed([13; 32])), but the index build is not. RaBitQ draws a random rotation at build time from an unseeded entropy RNG (random_fast_rotation_signs / random_orthogonal), and for nlist=4 the IVF k-means init is unseeded as well. One rotation is drawn per build and shared by every query, so recall varies run to run around the 0.5 bar. With 1-bit RaBitQ over uniform-random 32-dim data the mean recall sits at ~0.5, so the assertion is inherently borderline (the test already carried a "RQ doesn't perform well for random data" note and a commented-out #[ignore]).

Fix

Keep the strict >= 0.5 assertions and remove the flakiness by making the build deterministic through existing build-param seams, with no production change. The rotation is pinned via the existing RQBuildParams.rotation field (the same seam distributed builds use to share a rotation): the test constructs a fixed RabitQuantizationMetadata with fixed sign bits for the Fast rotation and a deterministic Haar-like orthogonal matrix (seeded Gaussian + Gram-Schmidt) for the Matrix rotation. The IVF centroids are pinned via IvfBuildParams::try_with_centroids, which bypasses k-means and removes the nlist=4 variance.

The change is confined to #[cfg(test)] mod tests. A no-op guard (determinize_rq_params returns params unchanged unless the last stage is RQ) leaves the SQ/PQ/flat/HNSW tests that share the same helpers unaffected.

Verification

All 12 test_build_ivf_rq permutations pass, including the previously-failing case_2::rotation_type_1_RQRotationType__Fast. Ran the matrix three times back-to-back with identical results. test_build_ivf_sq and test_build_ivf_pq still pass (shared-helper no-op path). cargo fmt --all and cargo clippy -p lance --tests -- -D warnings are clean.

@github-actions github-actions Bot added the chore label Jul 8, 2026
@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.54098% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance/src/index/vector/ivf/v2.rs 97.54% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@westonpace westonpace requested a review from BubbleCal July 8, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant